Evaluating User Experience in Technology Pilots

Computing devices and digital services have been moving rapidly from professional environments into the everyday life. This means that technology will influence the evolution of our everyday environment, including our physical surroundings, social encounters, and development of society. It becomes more and more important to evaluate the effects of technology in the realistic complex setting of everyday life in addition to controlled laboratory environments. In this paper, findings related to evaluating user experience in real-life trial conditions are summarized. The user experience evaluation methods are classified into four groups: (1) methods used before the pilot, (2) methods used during the pilot, (3) methods used immediately after the pilot and (4) follow-up studies. Each class bears their unique goals, possibilities and limitations for collecting user experience related data, and understanding it.